52 research outputs found
Argumentation Mining in User-Generated Web Discourse
The goal of argumentation mining, an evolving research field in computational
linguistics, is to design methods capable of analyzing people's argumentation.
In this article, we go beyond the state of the art in several ways. (i) We deal
with actual Web data and take up the challenges given by the variety of
registers, multiple domains, and unrestricted noisy user-generated Web
discourse. (ii) We bridge the gap between normative argumentation theories and
argumentation phenomena encountered in actual data by adapting an argumentation
model tested in an extensive annotation study. (iii) We create a new gold
standard corpus (90k tokens in 340 documents) and experiment with several
machine learning methods to identify argument components. We offer the data,
source codes, and annotation guidelines to the community under free licenses.
Our findings show that argumentation mining in user-generated Web discourse is
a feasible but challenging task.Comment: Cite as: Habernal, I. & Gurevych, I. (2017). Argumentation Mining in
User-Generated Web Discourse. Computational Linguistics 43(1), pp. 125-17
The Argument Reasoning Comprehension Task: Identification and Reconstruction of Implicit Warrants
Reasoning is a crucial part of natural language argumentation. To comprehend
an argument, one must analyze its warrant, which explains why its claim follows
from its premises. As arguments are highly contextualized, warrants are usually
presupposed and left implicit. Thus, the comprehension does not only require
language understanding and logic skills, but also depends on common sense. In
this paper we develop a methodology for reconstructing warrants systematically.
We operationalize it in a scalable crowdsourcing process, resulting in a freely
licensed dataset with warrants for 2k authentic arguments from news comments.
On this basis, we present a new challenging task, the argument reasoning
comprehension task. Given an argument with a claim and a premise, the goal is
to choose the correct implicit warrant from two options. Both warrants are
plausible and lexically close, but lead to contradicting claims. A solution to
this task will define a substantial step towards automatic warrant
reconstruction. However, experiments with several neural attention and language
models reveal that current approaches do not suffice.Comment: Accepted as NAACL 2018 Long Paper; see details on the front pag
Privacy-Preserving Models for Legal Natural Language Processing
Pre-training large transformer models with in-domain data improves domain
adaptation and helps gain performance on the domain-specific downstream tasks.
However, sharing models pre-trained on potentially sensitive data is prone to
adversarial privacy attacks. In this paper, we asked to which extent we can
guarantee privacy of pre-training data and, at the same time, achieve better
downstream performance on legal tasks without the need of additional labeled
data. We extensively experiment with scalable self-supervised learning of
transformer models under the formal paradigm of differential privacy and show
that under specific training configurations we can improve downstream
performance without sacrifying privacy protection for the in-domain data. Our
main contribution is utilizing differential privacy for large-scale
pre-training of transformer language models in the legal NLP domain, which, to
the best of our knowledge, has not been addressed before.Comment: Camera ready, to appear at the Natural Legal Language Processing
Workshop 2022 co-located with EMNL
DP-BART for Privatized Text Rewriting under Local Differential Privacy
Privatized text rewriting with local differential privacy (LDP) is a recent
approach that enables sharing of sensitive textual documents while formally
guaranteeing privacy protection to individuals. However, existing systems face
several issues, such as formal mathematical flaws, unrealistic privacy
guarantees, privatization of only individual words, as well as a lack of
transparency and reproducibility. In this paper, we propose a new system
'DP-BART' that largely outperforms existing LDP systems. Our approach uses a
novel clipping method, iterative pruning, and further training of internal
representations which drastically reduces the amount of noise required for DP
guarantees. We run experiments on five textual datasets of varying sizes,
rewriting them at different privacy guarantees and evaluating the rewritten
texts on downstream text classification tasks. Finally, we thoroughly discuss
the privatized text rewriting approach and its limitations, including the
problem of the strict text adjacency constraint in the LDP paradigm that leads
to the high noise requirement
Privacy-Preserving Graph Convolutional Networks for Text Classification
Graph convolutional networks (GCNs) are a powerful architecture for
representation learning on documents that naturally occur as graphs, e.g.,
citation or social networks. However, sensitive personal information, such as
documents with people's profiles or relationships as edges, are prone to
privacy leaks, as the trained model might reveal the original input. Although
differential privacy (DP) offers a well-founded privacy-preserving framework,
GCNs pose theoretical and practical challenges due to their training specifics.
We address these challenges by adapting differentially-private gradient-based
training to GCNs and conduct experiments using two optimizers on five NLP
datasets in two languages. We propose a simple yet efficient method based on
random graph splits that not only improves the baseline privacy bounds by a
factor of 2.7 while retaining competitive F1 scores, but also provides strong
privacy guarantees of epsilon = 1.0. We show that, under certain modeling
choices, privacy-preserving GCNs perform up to 90% of their non-private
variants, while formally guaranteeing strong privacy measures
Why do you think that? Exploring Faithful Sentence-Level Rationales Without Supervision
Evaluating the trustworthiness of a model's prediction is essential for
differentiating between `right for the right reasons' and `right for the wrong
reasons'. Identifying textual spans that determine the target label, known as
faithful rationales, usually relies on pipeline approaches or reinforcement
learning. However, such methods either require supervision and thus costly
annotation of the rationales or employ non-differentiable models. We propose a
differentiable training-framework to create models which output faithful
rationales on a sentence level, by solely applying supervision on the target
task. To achieve this, our model solves the task based on each rationale
individually and learns to assign high scores to those which solved the task
best. Our evaluation on three different datasets shows competitive results
compared to a standard BERT blackbox while exceeding a pipeline counterpart's
performance in two cases. We further exploit the transparent decision-making
process of these models to prefer selecting the correct rationales by applying
direct supervision, thereby boosting the performance on the rationale-level.Comment: EMNLP Findings 202
Trade-Offs Between Fairness and Privacy in Language Modeling
Protecting privacy in contemporary NLP models is gaining in importance. So
does the need to mitigate social biases of such models. But can we have both at
the same time? Existing research suggests that privacy preservation comes at
the price of worsening biases in classification tasks. In this paper, we
explore the extent to which this tradeoff really holds when we incorporate both
privacy preservation and de-biasing techniques into training text generation
models. How does improving the model along one dimension affect the other
dimension as well as the utility of the model? We conduct an extensive set of
experiments that include bias detection, privacy attacks, language modeling,
and performance on downstream tasks.Comment: Findings of ACL 202
The Legal Argument Reasoning Task in Civil Procedure
We present a new NLP task and dataset from the domain of the U.S. civil
procedure. Each instance of the dataset consists of a general introduction to
the case, a particular question, and a possible solution argument, accompanied
by a detailed analysis of why the argument applies in that case. Since the
dataset is based on a book aimed at law students, we believe that it represents
a truly complex task for benchmarking modern legal language models. Our
baseline evaluation shows that fine-tuning a legal transformer provides some
advantage over random baseline models, but our analysis reveals that the actual
ability to infer legal arguments remains a challenging open research question.Comment: Camera ready, to appear at the Natural Legal Language Processing
Workshop 2022 co-located with EMNL
One size does not fit all: Investigating strategies for differentially-private learning across NLP tasks
Preserving privacy in contemporary NLP models allows us to work with
sensitive data, but unfortunately comes at a price. We know that stricter
privacy guarantees in differentially-private stochastic gradient descent
(DP-SGD) generally degrade model performance. However, previous research on the
efficiency of DP-SGD in NLP is inconclusive or even counter-intuitive. In this
short paper, we provide an extensive analysis of different privacy preserving
strategies on seven downstream datasets in five different `typical' NLP tasks
with varying complexity using modern neural models based on BERT and
XtremeDistil architectures. We show that unlike standard non-private approaches
to solving NLP tasks, where bigger is usually better, privacy-preserving
strategies do not exhibit a winning pattern, and each task and privacy regime
requires a special treatment to achieve adequate performance.Comment: Accepted to EMNLP 2022; not a final camera-ready versio
To share or not to share: What risks would laypeople accept to give sensitive data to differentially-private NLP systems?
Although the NLP community has adopted central differential privacy as a
go-to framework for privacy-preserving model training or data sharing, the
choice and interpretation of the key parameter, privacy budget
that governs the strength of privacy protection, remains largely arbitrary. We
argue that determining the value should not be solely in the
hands of researchers or system developers, but must also take into account the
actual people who share their potentially sensitive data. In other words: Would
you share your instant messages for of 10? We address this
research gap by designing, implementing, and conducting a behavioral experiment
(311 lay participants) to study the behavior of people in uncertain
decision-making situations with respect to privacy-threatening situations.
Framing the risk perception in terms of two realistic NLP scenarios and using a
vignette behavioral study help us determine what thresholds would
lead lay people to be willing to share sensitive textual data - to our
knowledge, the first study of its kind
- …